FUNCTIONS ON SURFACES AND 
INCOMPRESSIBLE SUBSURFACES 



SERGIY MAKSYMENKO 

Abstract. Let M be a smooth connected compact surface, P be 
either a real line 1 or a circle S . Then we have a natural right 
action of the group D{M) of diffeomorphisms of M on C°°(M, P). 
For / S C°°(M,F) denote respectively by S(f) and 0(f) its sta- 
bilizer and orbit with respect to this action. Recently, for a large 
class of smooth maps / : M — > P the author calculated the ho- 
motopy types of the connected components of S(f) and O(f). It 
turned out that except for few cases the identity component of S(f) 
is contractible, KiO(f) — 7r,Af for i > 3, and 7T20(/) = 0, while 
iriO(f) it only proved to be a finite extension of 7Ti2?id(M) © l) 
for some I > 0. In this note it is shown that if %(M) < 0, then 
wiO(f) = G\ X • • • X G n , where each d is a fundamental group 
of the restriction of / to a subsurface Bi C M being either a 2- 
disk or a cylinder or a Mobius band. For the proof of main result 
incompressible subsurfaces and cellular automorphisms of surfaces 
are studied. 



1. Introduction 

Let M be a smooth compact connected surface and P be either the 
real line R or the circle S . Consider the right action of the group 
V{M) of diffeomorphisms of M on C°°(M, P) defined by 

h-f = foh- 1 

for h e V(M) and / G C°°(M, P). For every / G C°°(M, P) let 
0(/) = {/oM^D(M)}, 

5(/) = {/i| / = /oA, /iGP(M)} 

be respectively the orbit and the stabilizer of / with respect to this 
action. We will endow V(M), S(f), C°°(M,P), and 0(f) with the 
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corresponding topologies C°° . Denote by Sid{f) the identity path com- 
ponent of S(f) and by Of(f) the path component of / in 0(f). In [TU] 
the author calculated the homotopy types of «Sid(/) and Of(f) for all 
Morse maps / : M — > P. 

Moreover, in [12] the results of [10] were extended to a large class of 
maps with (even degenerate) isolated critical points satisfying certain 
"non-degeneracy" conditions. In fact there were introduced three types 
of isolated critical points (called S, P, and N) and the following three 
axioms for /: 

(Bd) / takes constant value at each connected component of dM and 
£/ C IntM. 

(SPN) Every critical point of f is either an S- or a P- or an N-point. 
(Fibr) The natural map p : T)(M) — >■ O(f) defined by p(h) — f o hT 1 
is a Serre fibration with fiber S(f) in topologies C°° . 

Recall that if / : (C, 0) — > (R, 0) is a smooth germ for which G C is 
an isolated critical point, then there exists a homeomorphism h : C — > 
C such that h(0) = and 



Examples of the foliation by level sets of / near are presented in 
Figure 11.11 



From this point of view S-points are saddles, while P- and N-points 
a local extremes. Moreover, P-points admit non-trivial /-preserving 
circle actions (as non-degenerate local extremes do), while N-points 
admit only Z n -action preserving /. We will not give precise definitions 
but recall a large class of examples of such points. 

Example 1.1. [10J. Let f : M 2 — > R be a homogeneous polynomial 
without multiple factors with degf > 2, so 




±|,z| 2 , if z is a local extremum, [3], 

Re(z n ), {n > 1) otherwise, so z is a saddle, [15], 




Figure 1.1. Isolated critical points 



f = L 1 ---L a -Q 1 ---Q b , a + 2b>2, 
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where every Li is a linear function and every Qj is an irreducible over 
R (i.e. definite) quadratic form such that Li/ Ly 7^ const for i ^ i' and 
Qj/Qji ^ const for j ^ f . 

If a — 1; so f has linear factors and thus is a saddle, then the 
origin G M 2 is an S -point for f . 

If a = and b = 1, so f = Q±, then the origin G M 2 is a P-point 
for f. 

Otherwise, a = and b > 2, so f = Q\---Qb- Then the origin 

G IR 2 is an N -point for f . 

Lemma 1.2. [TU]. Let / : M — > P be a C°° map satisfying (Bd) ; and 
such that every of its critical points belongs to the class described in 
Example \l.l\ in particular, f also satisfies (SPN). Then f also satisfies 
(Fibr). 

It follows from Morse lemma and Example ll.il that non-degenerate 
saddles are S-points while non-degenerate local extremes are P-points. 
Now the main result of [12] can be formulated as follows. 

Theorem 1.3. [JUIII2]- Suppose f : M P satisfies (Bd) and (SPN). 

If / has at least one S- or N-point, or if M is non-orientable, then 
$id(f) is contractible. 

Moreover, if in addition f satisfies (Fibr), then TtiOf(f) = 7TjM for 

1 > 3, 7r 2 0/(/) = 0, and for niO(f) we have the following short exact 
sequence 

1 -> TTiV{M) © ll TTiOf(f) G 1, 

for a certain finite group G and I > both depending on f . 

Thus, the information about the fundamental group TTiOf(f) is not 
complete. The aim of this note is to show that the calculation of 
niOf(f) can be reduced to the case when M is either a 2-disk, or 
a cylinder, or a Mobius band, see Theorems 11.71 and ll.8l below. The 
obtained results hold for a more general class of maps M — > P than 
the one considered in |12j . 

1.4. Admissible critical points. We will now introduce a certain 
type of critical points for /. Let F be a vector field on M, V <Z M be 
an open subset, and h : V — > M be an embedding. Say that h preserves 
orbits of F if for every orbit o of F we have that h(V fl o) C o. 

Definition 1.5. Let f : M P be a C°° map and z G IntM be 

an isolated critical point of f which is not a local extreme (so z is a 
saddle). Say that z is admissible if there exists a neighbourhood U 
of z containing no other critical points of f and a vector field F on U 
having the following properties: 
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(1) / is constant along orbits of F and z is a unique singular point 
ofF. 

(2) Let (Ft) be the local flow of F on U . Then for every germ 
of diffeomorphisms h : (M, z) — > (M, z) preserving orbits of 
F there exists a C°° germ a : (M, z) — > M such that h(x) = 
F(x, cr(x)) near z. 

This definition almost coincides with the definition of an S-point, 
c.f. [12]. The difference is that for S-points it is also required that the 
correspondence /z. i — >• cr is continuous with respect to topologies C°°. In 
particular every S-point is admissible. 

Now put the following two axioms for / both implied by (SPN): 

(Isol) All critical points of f are isolated. 
(SA) Every saddle of f is admissible. 

1.6. Main result. Let V^(M) be the identity path component of the 
group V(M) and 

5'(/)=5(/)nD id (M) 

be the stabilizer of / with respect to the right action of T>i^(M). Thus 
S'(f) consists of diffeomorphisms h isotopic to id.M and preserving F, 
i.e. / o h = f. 

For a closed subset X C M denote by S'(f, X) the subgroup of S'(f) 
consisting of diffeomorphisms fixed on some neighbourhood of X. 
The aim of this note is to prove the following theorem: 

Theorem 1.7. Suppose x{M) < 0. Let f : M — >■ P be a C°° map 
satisfying the axioms (Bd), (Isol), and (SA). Then there exists a compact 
subsurface X C M with the following properties: 

(1) / is locally constant on dX and every connected component B of 
M\X is either a 2-disk or a 2-cylinder or a Mobius band. Moreover, 
dB C X and B contains critical points of f . 

(2) Let h G S'(f,X) and B be a connected component of M\X, 
thus h is fixed on some neighbourhood of OB. Then the restriction h\s 
is isotopic in B to id# with respect to some neighbourhood of dB. 

(3) The inclusion i : S'(f,X) C S'(f) induces a group isomorphism 
i :ir S'(f,X)nir S'(f). 

The proof of this theorem will be given in £J71 We will now show how 
to simplify calculations of TC\0(f) using Theorem 1 1.71 

Let X be the surface of Theorem 11.71 and let Bi, . . . , B\ be all the 
connected components of M\X. For every i = denote by 
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Pid(Sj, dBi) the group of diffeomorphisms of Bi fixed on some neigh- 
bourhood of dBi and isotopic to id# 4 relatively to some neighbour- 
hood of Bi. Let also S'{f\B v dBj) be the stabilizer of the restriction 
f\ Bi '■ Bi P with respect to the right action of Vi^(Bi,dBi). Then 
we have an evident isomorphism of groups: 

(1.1) V X S'ifls^dBi), iP(h) = (h\ Bl ,...,h\ Bl ), 

i=l 

It is easy to show that ip is in fact a homeomorphism with respect to 
the corresponding C°° topologies. 



Theorem 1.8. Under assumptions of Theorem 1.1 suppose that f also 
satisfies (Fibr). Then we have an isomorphism: 

mOfif) « i TtaS'ifl^dBi). 

i=l 

Proof. It is easy to show that if / satisfies (Fibr), then Of(f) is the 
orbit of / with respect to the action of T)^(M) and the projection 
p : £>id(M) — > Of(f) is a Serre fibration as well, see [TT] . Hence we get 
the following part of exact sequence of homotopy groups 

► 7nAd(M) -> ttiO/C/) 7r 5'(/) TroPid(M) • • • 

Since *(Af) < 0, we have 7Ti£> id (M) = 0, [QUE]. Moreover > Ad(M) 
is path-connected, whence together with Theorem 11.71 we obtain an 
isomorphism: 

mOfif) « 7ro5'(/) ~ 7T 5'(/,X) ^ X TToS'iflB^dBi). 

i=l 

Theorem is proved. □ 

Thus a general problem of calculation of ■KiOf(f) for maps satisfying 
the above axioms completely reduces to the case when x(M) > 0. A 
presentation for iTiOf(f) will be given in another paper. 



1.9. Structure of the paper. In next four sections we study incom- 
pressible subsurfaces N C M. §2] contains their definition and some 
elementary properties. In §3] we show how such subsurfaces appear in 
studying maps M — > P with isolated singularities. In §2]and $5] we ex- 
tend results of W. Jaco and P. Shalen [8] about deformations of incom- 
pressible subsurfaces and periodic automorphisms of surfaces. £0 con- 
tains two technical statements about deformations of diffeomorphisms 
preserving a map M — > P. Finally in £j7]we prove Theorem 1 1.71 
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2. Incompressible subsurfaces 

The following Lemma [27T1 is well-known, see e.g. [14, Pr. 2.1]. It was 
also implicitly formulated in [HI page 359]. 

Lemma 2.1. 1) Let M be a connected surface, and N C IntM be 

a proper compact (possibly not connected) subsurface neither of whose 
connected components is a 2-disk. Then the following conditions are 
equivalent: 

(a) for every connected component of N the inclusion homomor- 
phism TtiNi — > -K\M is injective; 

(b) none of the connected components of M\N is a 2-disk. 

If these conditions hold, then N will be called incompressible, 
see Def. 3.2]. 

Corollary 2.2. If N C M is incompressible, then x{M) < x(N)- 

Corollary 2.3. Let R C IntM be a proper compact connected subsur- 
face. Then the following conditions are equivalent: 

(Rl) the homomorphism £ : tciR — > i\\M is trivial; 
(R2) R is contained in some 2-disk D C M. 

Proof. The implication (R2)=^>(R1) is evident. 

(R1)=^>(R2). Suppose R is not contained in any 2-disk. We will show 
that £ is non-trivial. Let N be the union of R with all of the connected 
components of M \ N which are 2-disks. Then by our assumption N 
is not a 2-disk and by Lemma [27T1 iV is incompressible. Notice that £ is 
a product of homomorphisms induced by the inclusions R C N C M: 

£ = p o a : KiR 4 TC\N A TTiM. 

Also notice that a is surjective and by Lemma I2TT1 is a non-trivial 
monomorphism. Hence £ is also non-trivial. □ 

Corollary 2.4. Let R C IntM be a proper (possibly non connected) 
subsurface such that neither of its connected components is contained 
in some 2-disk. Then every connected component B of M\R which is 
not a 2-disk is incompressible. 

Proof. Let C be a connected component of M \ B. Due to Lemma [27T1 
it suffices to show that C is not a 2-disk. Notice that CDR ^ 0, whence 
it contains some connected component Ri of R. By Corollary 12.31 the 
product of homomorphisms n\Ri — > iiiC — > Ti\M is non-trivial, and 
therefore i\\C — > -K\M is also non-trivial. This implies that C is not a 
2-disk. □ 
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3. Incompressible subsurfaces associated to a map M -> P 

3.1. Singular foliation Af of /. Let / : M — > P be a map satisfy- 
ing axioms (Bd) and (Isol). Then / induces on M a one-dimensional 
foliation A f with singularities defined as follows: a subset u C M is a 
leaf of Af if and only if u is either a critical point of f or a connected 
component of the set f~ 1 (c)\Y,f for some c G P. Thus the leaves of Af 
are 1-dimensional sub manifolds of M and critical points of /. Local 
structure of Af near critical points of / is illustrated in Figure HTT1 

Denote by Ay eg the union of all leaves of Af homeomorphic to the 
circle and by AJ the union of all other leaves. The leaves in A^ eg 
(resp. AJ) will be called regular (resp. critical). Similarly, connected 
components of A^ eg (resp. A") will be called regular (resp. critical) 
components of Ay. It follows from (Bd) that dM C A^. eg . It is also 
evident, that every critical leaf of A? either is homeomorphic to an 
open interval or is a critical point of /. 

3.2. Atoms and canonical neighbourhoods of critical compo- 
nents of Af. For every critical component K of Af define its regular 
neighbourhood Rk as follows. Let ci, . . . , q be all the critical values 
of / and the values of / on dM. Since M is compact, it follows from 
axioms (Bd) and (Isol) that I is finite. For each i — 1, . . . , I let W{ C P 
be a closed connected neighbourhood (i.e. just an arc) of q containing 
no other Cj. We will assume that W{ R Wj = for i ^ j. 

Now let K be a critical component of Af. Then f(K) = C{ for 
some i. Let Rk be the connected component of / _1 (Wi) containing K. 
Evidently, Rk is a union of leaves of Af. Following [2] we will call Rk 
an atom of K, see Figure [3TTT . 




Figure 3.1. 

Evidently, Rk is a regular neighbourhood of K with respect to some 
triangulation of M. Similarly to [8] define the canonical neighbourhood 
Nk of K to be the union of Rk with all the connected components of 
M \ Rk being 2-disks. If iVV is not a 2-disk, then by Lemma 12.11 Nk 
is incompressible in M. 

Notice that 



(3.1) 



dR K = f-\dWi) n R K . 
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Let K' be another critical component of Af such that f(K') = f(K). 
Since Rk< is also constructed via Wi, we obtain from (13.11) that / takes 
on 8Rk' the same values as on 8Rk ■ This technical assumption is not 
essential, however it will be useful for the proof of Theorem 1 1.71 

Lemma 3.3. Let K and K' be two distinct critical components of Af. 

(i) Then R K D Rk> = 0, while N K and N K > are either disjoint or 
one of them, say Nk, is contained in Nk<- In the last case Nk 
is a 2-disk. 

(ii) Suppose f(K) = f(K') and there exists h G S(f) such that 
h{K) = K'. Then h{R K ) = R K , and h{N K ) = N K >. 

Proof, (i) follows from the assumption that Wi D Wj = for i ^ j, 
and (ii) follows from (13.11) . We leave the details for the reader. □ 

Lemma 3.4. Let K be a critical component of Af such that Nk is a 
2-disk. Then either 

(i) M is a 2-disk itself, or 

(ii) Nk is contained in a unique canonical neighbourhood Nk> of 
another critical component K' of Af such that Nk 1 is not a 
2-disk. 

Proof. Let R be the union of atoms of all critical components of Af. 
Then every connected component B of M \ R is diffeomorphic to the 
cylinder S 1 x [0, 1] and the restriction f\s has no critical points. 

Notice that M \ Nk is connected since Nk is a 2-disk. Also, there 
exists a unique connected component B (being a cylinder S 1 x [0, 1]) 
of M\ R such that dN K C B. Then N K U B is also a 2-disk. 

Let n be the total number of critical components of Af in M \ Nk- 

If n = 0, then N K UB = M. Whence M is a 2-disk. 

Suppose that n > 1. Let 7 be another connected component of dB 
distinct from 8Nk- Then there exists an atom Rk 1 of some critical 
component K' of Af such that 7 C 8Rk'- Since Nk^B is a 2-disk, we 
see that it is contained in Nk 1 - If Nk 1 is not a 2-disk, then the lemma 
is proved. Otherwise, the number of critical components in M \ Nk 1 is 
less than in M \ Nk and the lemma holds by the induction on n. □ 

Example 3.5. Let T 2 be a 2-torus embedded in R 3 as shown in Fig- 
ure 13.21 and / : T 2 — > K be the projection onto the vertical line. Fig- 
ure 13.2b ) shows the critical components of level-sets of /, and Fig- 
ure [32b) presents blackened canonical neighbourhoods of three critical 
components of Af containing canonical neighbourhoods of all other 
critical components of Af. 
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a) b) 
Figure 3.2. 

3.6. Canonical neighbourhoods of negative Euler character- 
istic. Suppose M is not a 2-disk. Let Ki,...,K r be all the crit- 
ical components of Af whose canonical neighbourhoods are not 2- 
disks. By Lemma [33] this collection is non-empty and by Lemma I3U1 
Nx- fl Nk. = for i 7^ j. Moreover, again by Lemma [331 an y other 
critical component of Af is contained in some Nk { - It follows that 
M \ Ul—iNjCi contains no critical points of /, whence it is a disjoint 
union of cylinders S 1 x /. Therefore 

r 

(3-2) x(AO = E*)- 

8=1 

The following two statements will be used for the construction of a 
surface X of Theorem 11.71 see £J7[ 

Lemma 3.7. The following conditions are equivalent: 

(1) X (M) < 0; 

(2) xi^Ki) < for some i — 1, . . . , r. 

Proof. (1)=K2). As x(M) < 0, we get from that x{N Kl ) < for 
some i. 

The implication (2)=^(1) follows from Corollary 12.21 □ 

Corollary 3.8. Let Ki, . . . , K% be all the critical components of Af 
whose canonical neighbourhoods have negative Euler characteristic and 
Rk 1 , • • • , Rx k be their atoms. Put 7£<o := Uf =1 i?Xi- If ^<o 7^ , 
then every connected component B of M\ 7^<o is either a 2-disk, or a 
cylinder, or a Mobius band. 

Proof. Since the homomorphism 7Ti-RfQ — >* ttiM is non-trivial for each 
i, it follows from Corollary 12.41 that B is incompressible. Suppose 
x{B) < 0. Notice that / takes constant values of OB. Then by 
Lemma I3TT1 there exists a critical component K C B of Af such that the 
canonical neighbourhood N of K with respect to f\s has negative Euler 
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characteristic. It follows that the homomorphisms 7TiiV — > ttiB — > t\\M 
induced by the inclusions N C B C M are monomorphisms, so N is 
incompressible in M. This implies that N is a canonical neighbour- 
hood of K with respect to /. But since x(iV) < 0, we should have that 
iV C 7^ <0 , which contradicts to the assumption. □ 

4. Deformations of incompressible subsurfaces 

The aim of this section is to extend some results of [8] concerning 
incompressible subsurfaces, see Proposition 14.51 

4.1. ±-twist. Let 7 C IntM be a two-sided simple closed curve, U 
be its regular neighbourhood diffeomorphic to S 1 x [—1, 1] so that 7 
correspond to S 1 x 0. Take a function /i : [—1, 1] — » [0, 1] such that 
fi = near {±1} and /x = 1 on some neighbourhood of 0. Define the 
following homeomorphism //- : M — > M by 



(4.1) 9l (x) 



(ze 27Ti ^,t), x = (z,t) E S 1 x [—1, 1] = U 



xeM\U, 



see Figure EL~T1 Then is fixed on some neighbourhood of M \ U and 
isotopic to id.M via an isotopy supported in IntU. Evidently, g 7 is a 
product of Dehn twists in opposite directions along the curves parallel 
to 7. Therefore we will call g 7 a ±-twist near 7. 



Figure 4.1. ±- twist 

The following lemma is a particular case of [HI Lm. 6.1]. 

Lemma 4.2. [61 Lm. 6.1]. Suppose x(M) < 0. Let 7 C IntM be a 

simple closed curve which does not bound a 2-disk nor a Mobius band, 
h : M — >■ M be a homeomorphism homotopic to idjv/ and such that 
^(7) = 7- Let also H : M x / — )■ M fre any homotopy of id m to ^> 
T/ien t/iere exists another homotopy Gt '■ M x / — > M 0/ id^ to /i s«c/i 
i/iat G t (7) = 7 and G 4 = on M\U for all t G /. 

Moreover, there exists m G Z and a homotopy G' : M x J — >■ M 0/ 
id A/ to g™ oh such that G' t = G outside U and G' t is fixed on 7 for all 
t G /. 

The following statement is also well-known. 
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Lemma 4.3. Let M be a surface with x(M) < 0. Suppose dM ^ 
and let 7 1 ,...,7j fre a// the connected components of DM. For each 
i = 1, . . . , I letTi be a Dehn twist along the curve parallel to 7, and fixed 
on dM. Let m\, . . . ,mi G Z be integer numbers not of all are equal to 
zero. Then the homeomorphism t™ 1 o ■ ■ ■ o t™ 1 is not homotopic to 
id.M via a homotopy fixed on dM. 

4.4. Deformations of incompressible subsurfaces. Let M be a 

surface distinct from the 2-sphere S 2 and the projective plane RP 2 , 
N C M be an incompressible subsurface, and Ni, . . . , be all of its 
connected components. Let also h : M — > M be a homeomorphism 
homotopic to id.M and H : M x I — > M be any homotopy of id^ to h. 

The following Proposition 14.51 follows the line of [H Lm. 4.2]. In fact 
the first part of statement (B) is a particular case of that lemma. 

Proposition 4.5. c.f. [H Lm. 4.2] (A) If Nj is not a cylinder for some 
j, then h(Nj) n Nj ^ 0. 

(B) Suppose x{Nj) < and h(Nj) C Nj for some j. Then there 
exists a homotopy G : Nj x I — >■ Nj of the identity map id^. to the 
restriction h\jf- such that G t {x) = H t (x) whenever H(x x J) c Nj. 

Moreover, suppose x /) c 7 for each connected component 7 of 
dNj. Extend G to a map G : M x I ^ M by G t = H t on M \ Nj. 
Then G is a homotopy of id m to h. 

(C) Suppose x(Nj) < and h(Nj) = Nj for all j = 1, . . . , k. Then 
there exists a homotopy G : M x / — > M o/idjvf to h such that G(Nj x 
J) C Nj for all j = 1, . . . , k and G(B X I) C B for every connected 
component B of M\N. 

(D) Suppose x(Nj) < and h is fixed on N for all j = 1, . . . , k. 
Then there exists a homotopy of id m to h fixed on N . 

Proof. First we make the following remark which repeats the key ar- 
guments of [51 Lm. 4.2]. For j = 1, . . . , k let pj : Mj — > M be the 
covering map corresponding to the subgroup 7r\Nj of it\M. Then the 
embedding i : Nj C M lifts to the embedding i* : Nj — > Mj which 

induces an isomorphism between KiNj and TTiMj. Denote Nj = i*(Nj). 
Then we have the following commutative diagram: 
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Since Mj and Nj are aspherical, it follows from Whitehead's theorem 
that Nj is a strong deformation retract of Mj. Then every connected 

component of Int (Mj \ Nj) is an open cylinder. Let H : Nj x I —t M 
be any homotopy between the identity embedding Hq = i : Nj C M 
and H\ = /i|jv r Then there exists a lifting H : iVj x I — >■ Mj such that 
Jf = and PjoH = H. Denote iVj = #i(iVj). Since both iVj and iVj 
are deformational retracts of Mj, they are incompressible in Mj. 

(A) Suppose h(Nj) fl iVj = 0. Then Int(iVj) is included into some 

connected component C of Int(Mj \ Nj) being a cylinder. Since Nj 

is incompressible in M, it is also incompressible in C, whence N'- and 
therefore Nj are cylinders. Thus if Nj is not a cylinder, then we obtain 
that h(Nj) n Nj ^ 0. 

(B) Let Tj : Mj — >■ iVj be any retraction. Then the map 

G = pj o r j o H : iVjX/ -> Nj 

is a homotopy of id^. to h\^ j in A^-. It is easy to see that G t (x) = H t (x) 
whenever H(x x /) c Nj. 

Suppose that H{j x I) C 7 C Nj for each connected component 7 
of <9iVj. Then by the construction G t = H t on dNj. Notice that dNj 
separates M. Extend G to all of M x J by G = H of (M \ iVj) x J. 
Then G is continuous, G = id*/ and G*i = h. 

(C) Suppose x(-Aj) < and h(Nj) = Nj for all j = 1, . . . , k. Let 
71, . . . ,7; be all the connected components of dN. Since iV is incom- 
pressible, we have by Corollary 12. 21 that x{M) < x{Nj) < as well. 
Moreover, by (B) for each j the restriction /i|jv. is a homeomorphism of 
Nj homotopic in Nj to idjv ■ This, in particular, implies that h(ji) = ji 
for i — 1, . . . , I. 

Then by Lemma H~2l we can suppose that H(ji x /) c 7; for all 
i — 1, . . . , I as well. Moreover, due to (B) it can be additionally assumed 
that H(Nj x J) c iVj. 

Let B be a connected component of M\N. Since iV is incompress- 
ible, B is not a 2-disk. Then by Corollary 12.41 B is incompressible. 
Therefore we can apply statement (B) to B and change the homotopy 
G on B x I so that G(£ x J) c B. 

(D) Suppose ft, is fixed on N. For each i let C7i be a regular neigh- 
bourhood of 7i, and g>j be a ±- twist near 7^ supported in t/j. We can 
assume that [/, D Uj = for i ^ j. Then by Lemma 14.21 there exist 
integer numbers mi, . . . , mi 6 Z and a homotopy G : M x 7 — >■ M of 
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id.M to a homeomorphism h! := g™ 1 o • • • o g™ 1 o h such that Gt is fixed 
on L for each t G /. By (C) we can also assume that G(Nj x J) c iVj 
and x I) c B for every connected component of M \ A" and each 
J = l,...,fc. 

In particular, we see that the restriction h'\^ is homotopic to idAr 
relatively dN. But this restriction is evidently a product of Dehn twists 
along boundary components of N. Since xi-^j) < f° r a ^ h we S e t 
from Lemma 14^31 that rri{ = for all i = 1, . . . , I. Hence h! = h. Thus 
G is in fact a homotopy between id^ and h relatively dN. Since dN 
separates M, and idjv/ and /i are fixed on N, we can change G on N x I 
by = x. This gives a homotopy between id^ and h relatively to 

N. □ 

5. Automorphisms of cellular subdivisions 

Let iV be a compact surface and H = {ca}aga be some partition 
of into a disjoint family of connected orientable submanifolds. Say 
that a homeomorphism h : A^ — > N is a ^-homeomorphism provided it 
yields a permutation of elements of 5, that is for each e G H its image 
/i(e) also belongs to 5. An element e G 5 will be called h-invariant if 
fo(e) = e. Say that e is h + -invariant {h~ -invariant) if the restriction 
h\ e : e — > e is a preserving (reversing) orientation map. We will also 
say that h is "E-trivial if each e G 5 is /i + -invariant. 

Remark 5.1. Notice that we can say that a map h : e — > e preserves or 
reverses orientation only if dime > 1. To each O-dimensional element 
e G H (being of course a point) we formally assign a "positive orien- 
tation" and assume that by definition every cellular homeomorphism 
preserves orientation of each invariant 0-element of S. 

Example 5.2. Let M be a connected surface and K C IntM be an 
embedded finite connected graph. Assume that AT is a subcomplex of 
M with respect to some triangulation of M. By Rk we will denote a 
regular neighbourhood of K. Following [8] define a canonical neigh- 
bourhood Nk of A' to be the union of a regular neighbourhood Rk of K 
with those connected components of M \ Rk which are 2-disks. Notice 
that Nk \ K is a disjoint union of open 2-disks and half-open cylinders 
S 1 x (0, 1] with S 1 x {1} corresponding boundary components of 8Nk- 
Thus we obtain a natural partition of A^ by vertexes and edges of K 
and connected components of A^ \ K. We shall denote this partition 
by Ek- 

Now let H be a cellular subdivision of N. Denote by Aj (i = 0, 1, 2) 
the z-th skeleton of N. In particular, N% is a finite connected subgraph 
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in N such that N \ N\ is a disjoint union of 2-disks. Let c$ (i = 0, 1, 2) 
be the total number of i-cells of A. Then of course x(N) = c — C\ + c 2 . 

Let C = {Cj, di} be the M-chain complex of N corresponding to a 
given cellular subdivision. Thus Cj is a real vector space of dimension q 
with the oriented z-cells of H as a basis. Then every S-homeomorhism 
h induces a chain automorphism {hi : Cj — > Ci, i = 0, 1, 2} of C. 

Recall that for each continuous mapping h : iV — > N we can define 
its Lefschetz number L(h) by the formula: 

L(h) = tr(ft ) -tr(^) +tr(^ 2 ), 

where ^ : Hi(N,~R) — > Hi(N,Wj is the induced homomorphism of the 
corresponding homology groups and tr is the trace of this homomor- 
phism. If ft, is cellular, then can also be calculated via the chain 
homomorphisms hi by: 

L(h) = tr(/io) - tr(fti) + tr(/i 2 ). 

The following theorem is relevant to [HJ Lm. 4.4] being a statement 
about periodic homeomorphisms. 

Theorem 5.3. c.f. [8 : , Lm. 4.4]. Let M be a compact surface, K C M 
a connected subgraph, Nk be a canonical neighbourhood of K . Let 
also h : M — >■ M a homeomorphism such that h is homotopic to id^? 
h(K) = K , and h preserves the set of vertexes of K of degree 2, and 
h(Nx) = Nk- In particular, h\^ K is a -homeomorphism. 

(1) If x(Nk) < 0, then h is Ex-trivial. 

(2) Suppose that Nk = M , M is orientable, and x{M) > 0. Then 
every annulus a G is h + -invariant, and the total number 
of h-invariant cells o/H^ is equal to x(M). 

The proof of Theorem 15.31 will be given in §5.71 It is based on Propo- 
sition 075] and on the following statement. 

Proposition 5.4. Let N be a closed, orientable surface endowed 
with some cellular subdivision E and h : N — )■ iV be a E-homeomorphism 
preserving orientation of N and being not E-trivial, i.e. h(e) ^ e 
for some cell e G E. Then the number of h-invariant cells of E is 
precisely equal to L(h). In particular, L{h) > 0. 

Proof. Let fcj (i = 0, 1, 2) be the number of h- invariant z-cells of 5 and 
k := ko + k\ + fc 2 . We will show that 



(5.1) 



h = (-1) 1 tr(^), 
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which will imply 

2 2 
i=0 i=0 

To prove (15. ip we have to show that there are no /^"-invariant 0- 
and 2-cells and no /i + -invariant 1-cells. For 0-cells this holds by Re- 
mark [5J] and for 2-cells from the assumptions that N is orientable and 
h preserves orientation. 

Let e be an /i-invariant 1-cell and fo and f\ be two 2-cells that are 
incident to e. It is possible of course that f = f\. Since h preserves 
orientation, it follows that 

(a) either h 2 (fj) = +fj for j = 0, 1, and hi(e) = +e, 

(b) or h 2 (fj) = +fi-j for j = 0, 1, and hx{e) = -e. 

The following Claim 15.51 implies that in the ) h is S-trivial. 

Since h is not S-trivial, we will get from (b) that all /i-invariant 1-cells 
are h~ -invariant. 

Claim 5.5. Suppose that there exists a 1-cell e G H such that 

(i) hi(e) = +e G C\ and 

(ii) h preserves each 2-cell which is adjacent to e. 
Then h is ^-trivial. 

Proof. Notice that for each vertex v G -/V the inclusion N\ C iV induces 
a cyclic ordering of edges that are incident to v. 

Let v be a vertex of e. Then it follows from (i) and (ii) that all of the 
1- and 2-cells incident to v are /i + -invariant. Moreover, for each 1-cell 
that is incident to v the conditions (i) and (ii) also hold true. Since N 
is connected, it follows that h is S-trivial. □ 

Proposition 15.41 is completed. □ 

Corollary 5.6. Let N be a closed surface, H be a cellular subdivision 
of M, and h : iV — >■ iV be a S-homeomorphism. If h is isotopic to idjv ; 
then each of the following conditions implies that h is E-trivial: 

(1) \i.Y)-. 0: 

(2) x{N) > and the total number of h + -invariant 2-cells is greater 
than x(N). 

Proof. Since h is isotopic to idjv, we have that L(h) = L(idAr) = x(iV). 

If iV is orientable, then h preserves orientation and by Proposition l5.4l 
h is either S-trivial or has exactly x(iV) > invariant cells. Each of 
the conditions (1) and (2) implies that the number of /i-invariant cells 
is not equal to x(iV). Hence h is S-trivial. 
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Suppose that N is non-orientable and let p : N — > N be its oriented 
double covering. Then 5 lifts to some cellular subdivision 5 of N and 
h lifts to a unique S-cellular homeomorphism h of N which is isotopic 
to id#. Therefore L(h) = L(id^) = x(N) = 2x(iV). 

We claim that every of the conditions (1) and (2) implies that h is 
S-trivial, whence h will be S-trivial. 

(1) If x(N) < 0, then x{N) < 0, whence h is 5-trivial. 

(2) Suppose that x(N) > and the total number b of /i + -invariant 
2-cells is greater than xO^O- Let e be an /i + -invariant 2-cell of 5 and 
ei and e~2 be its liftings in 5. Then they are /^-invariant. Hence h has 
at least 2b > 2x{N) = x(N) invariant cells. Then by Proposition 15.41 
h is H-trivial. □ 



5.7. Proof of Theorem 15. 3L Let h : M — > M be a homeomorphism 
homotopic to the identity and such that H\n k is a H^-homeomorphism. 
Let 71, . . . , 7b be all the connected components of ONk, and a±, . . . , a& 
be the annuli of Ex corresponding to them, so that 7, C cij. Shrink 
every 7$ to a point Xi and denote the obtained surface by Nr. Then 
Nk is a closed orientable surface and yields an evident cellular 
partition 5 of Nk such that each annulus <Zj corresponds to a certain 
2-cell <ij G 5. 

Also notice that x(A^r) = x(-Na-) + b. 

Claim 5.8. Suppose that either x{Nk) < or Nk = M. Then 

(a) H\n k is homotopic to idjy K in Nk- 

(b) /i(7i) = 7i /or z = 1, . . . , b and h preserves orientation 0/7^; 

(c) fa induces some E-homeomorphism h : N K — )■ A^ homotopic 
to idjy^ wii/i respect to {x±, . . . ,Xb}, in particular, every 2-cell 

cii G E is h + -invariant; 

(d) L(fa) = L(id^) = x(iVx) = x(JVjc) + 6- 

Proof, (a) For A# = M this statement is trivial. If x{Nk) < 0, then by 
(B) of Proposition 14.51 (or directly by [El Lm. 4.1]) h\^ K is homotopic 
to idjvjf in iV^. All other statements (b)-(d) follow from (a). □ 

Now we can complete Theorem 15. 31 

(1) Suppose that x( n k) < 0. If also x( n k) < 0, then by (1) of 
Corollary 15.61 h is S-trivial, whence h is S^-trivial as well. 

Let x(Nk) — 0. By Claim [5T81 h has at least b /i + -invariant 2-cells 
ai, . . . , aft. Moreover, since x{Nk) — b = x(Nk) < 0, we obtain that 
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b > x(iVft-), whence by (2) of Corollary 15.61 h is 5-trivial. Therefore h 
is 5^-trivial. 

(2) Suppose that Nk = M and M is orientable. It follows from (c) of 
Claim 15.81 and Proposition 15.41 that h is either H-trivial or has exactly 
x{Nk) invariant cells. Therefore, h is either 5^-trivial or has exactly 
x{Nk) — b = x(Nk) = x{M) invariant cells. 

6. Deformations of diffeomorphism near critical 
components of a/ 

The following two propositions will be crucial for the proof of The- 
oremO Suppose / : M P satisfies (Bd), (Isol), and (SA). 

Proposition 6.1. Let K be a critical component of Af such that every 
z G K flSj is admissible, R be its atom, and U be any neighbourhood 
of R. Let also h G S(f). Suppose that h(u>) = u for each leaf u 
of Af contained in K and that h preserves orientation of u whenever 
dim a; = 1. Then h is isotopic in S(f) to a diffeomorphism h! G S(f) 
such that h! = h on M\U, and h! is the identity on some neighbourhood 
of R in U. 

Proof. This proposition follows the line of [101 Th. 6.2]. For the con- 
venience of the reader we will recall the key arguments for the case 
when M is orientable. A non-orient able case can be deduced from the 
orientable one similarly to [121 Th. 6.2]. 

As M is orientable, it has a symplectic structure. Let H be the 
Hamiltonian vector field of /. Then / is constant along orbits of H, 
the set of singular points of H coincides with the set of critical points 
of /, and the foliation by orbits of H coincides with A^. In particular, 
H is tangent to DM and therefore generates a flow H : M x M. — > M. 

We will now change H on neighbourhoods of admissible critical 
points of / similarly to [TUJ Lm. 5.1]. Let z G £/ be such a point 
and F z be a vector field on some neighbourhood U z of z satisfying as- 
sumptions of Definition [T3J Then it follows from (i) of Definition [T75] 
that for every x G U z the vectors H(x) and F z are parallel each other. 
Therefore, using partition unity technique and changing (if necessary) 
the signs of F z , we can change H near each z G R D £ / and assume 
that H = F z on U z . 

Claim 6.2. There exists a neighbourhood U of R and a unique C°° 
function a : U — > R such that h(x) = H(x, cr(x)) for all x G U . 

Proof. Let z G K R £/. By assumption h preserves leaves of Af (i.e. 
orbits of H) in K with their orientations. Since F z = H near z, it fol- 
lows from (ii) of Definition II. 51 that there exists a neighbourhood V z of 
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z and a unique C°° function a z : V z — > R such that h(x) = H(x, a z (x)). 
Then the functions {o" z } z6 i S - n s / yield a unique C 00 function a on the 
union U V z . It remains to note that .K" \ is a disjoint union of 



open intervals, whence a uniquely extends to a C°° function on R such 



Then the desired isotopy of h to h! in S(f) can be constructed simi- 
larly to [10j Lm. 4.14]. Take any C°° function // : M — > [0, 1] such that 
/i = on some neighbourhood of M \ ?7, fi = 1 on i?, and \i is constant 
along orbits of F. Then the function v = fia is C°° and well-defined 
on all of M. Consider the following homotopy 

(6.1) g : M x / M, ^(.x) = F(x, ti/(ar)). 



Then ^ = idj^, fl 1 * is fixed on M\U, and = h on i?. Since // 
is constant along orbits of F and h is a diffeomorphism, it follows 
from [TQl Lm. 4.14] that g is an isotopy. Hence g^ 1 o h : M — > M, 
(t G J), is an isotopy in «S(/) supported in U and deforming ft to a 
desired diffeomorphism h! = g^ 1 oh. □ 

Proposition 6.3. Let X C M be a compact subsurface such that dX 
consists of (regular) leaves of Af. Suppose h G Sid(f) is fixed on some 
neighbourhood U of X . Then there exists an isotopy of h to idM in 
S(f) fixed on some neighbourhood of X. 

Proof. Again we will consider only the case when M is orientable. Let 
H : M x R — > M be the flow constructed in the proof of Proposition 16.11 
Since h G Sid(f), there exists an isotopy G : M x / — y M of id^j to 
h in S(f). Then is it easy to show that each G t preserves orbits of 
H on some neighbourhood of X, see [10, Lm. 3.4]. Now it follows 
from Th. 25], see also [13], that there exists a continuous function 
A:(M\S/)x/->R such that A t is C°° for each t G /, A = 0, and 
G t {x) = H(x,A t (x)) for all x G M \ £/. Let n : M -»■ [0, 1] be a C°° 
function constant along orbits of if, /x = on X, and /i = 1 on some 
neighbourhood of M \ U. Define the following map a : M x I — > M by 



We claim that a is an isotopy between idj^ and h in S(f) fixed on some 
neighbourhood of X. 




□ 
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Since /i = 1 on some neighbourhood of M\U, we see that a is 
continuous and a t is C°° for each t. Moreover, 



a(x, 0) 



H(x,0) = x, xeU 
G (x)=x, xeM\U. 



Since h is fixed on U, it follows that A(x, 1) = on U . Therefore 
fiAi = Ai and a± — h. As fi = on X, we obtain that a t , (t £ /), is 
fixed on X. □ 



7. Proof of Theorem 11.71 

Suppose x(M) < and that / : M ->■ P satisfies (Bd), (Isol), and 
(SA). We have to find a compact subsurface X C M satisfying condi- 
tions (l)-(3) of Theorem 11.71 

Construction of X. Let X 1; . . . , ii^ be all the critical components of 
level-sets of / whose canonical neighbourhoods have negative Euler 
characteristic: x(^Ki) < 0. Since x(^0 < 0, we have by Lemma [33 
that this collection is non-empty. Denote 

K = U Ki, 

i=l 

For each % = 1, . . . , k choose an atom Pj for in a way described 
in §3.2[ and let iVj be the corresponding canonical neighbourhood of 
Ki. Then we can assume that conditions (i) and (ii) of Lemma [3731 
hold. In particular, Pj D Rj = iVj PI iVj = for z ^ j. 

Denote 7^. <0 := U P«. Let also Bi,...,B q be all the connected 

i=l 

components of M \ TZ<o such that every P« is a cylinder and / has no 
critical points in Pj. Put 

X = 1Z <0 U Pi U • • • U B q . 

We will show that X satisfies the statement of Theorem 11.71 

Example 7.1. Let M be an orientable surface of genus 2 embedded in 
M. 3 in a way shown in Figure ITTTk ) and / : M — > R be the projection to 
the vertical line. Critical components of level-sets of / whose canonical 
neighbourhoods have negative Euler characteristic are denoted by K\ 
and K2. The corresponding surface X is shown in Figure 1771b ). 

Before proving Theorem II .71 we establish the following statement. 

Claim 7.2. (i) Let h G S'(f). Then h preserves every leaf u C "P<o 
of Af and its orientation. 
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L 




Li !\; \ 




a) 



Figure 7.1. 



(ii) Suppose h is fixed on a neighbourhood of TZ<o- Then for every 
connected component B of M \1Z <0 the restriction H\b is isotopic to 
ids with respect to a neighbourhood of OB fl 7£<o. 

Proof, (i). It follows from the definition of K. that h(JC) = /C. We claim 
that in fact h(Ki) = Ki for alH = 1, . . . , k. 

Indeed, suppose that h(Ki) = Kj for some j. Then by Lemma [3731 
h(Ri) = Rj and h(N{) = Nj. On the other hand, since N is incom- 
pressible, x(iVj) < 0, and h is isotopic to idjvf , it follows from (1) of 
Proposition S3] that h(Ni)nNi ^ 0. But NnNj = for % ^ j. Hence 
h(Ni) = Ni for each % = 1, . . . , k. 

Denote by Hj the corresponding partition of Ni, see §3 Since h 
preserves the set of critical points of /, it follows that h preserves the 
set of vertexes of degree 2 of Ki. This implies that the restriction 
of h to Ni yields a certain automorphism h* of the partition Sj. As 
x{N) < and h is isotopic to id^, we get from Theorem 15.31 that h 
yields a trivial automorphism of Hj. In particular, each (critical) leaf 
uj of Af in Ki is /i + -invariant. 

Let u C Ri be a regular leaf of and e C N be the corresponding 
element of H, containing w, so e is either an open 2-disk or a half-open 
cylinder S 1 x (0,1]. Then 



Notice that h(e) = e, since h is Hj-trivial. Moreover, foh — f implies 
that h o o f(u) = f^ 1 o f(oj), whence h(u) = uj. It remains to note 
that h preserves orientation of u since it preserves orientation of leaves 
in Ki. 

(ii) Let B be a connected component of M \ IZ^. Then it follows 
from Corollary 13.81 that B is either 



e n / 



o 
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(a) a 2-disk, or 

(b) a Mobius band, or 

(c) a cylinder such that one of its boundary components belongs to 
7Z <0 and another one to dM, or 

(d) a cylinder with dB C 7L<q. 

If B is of type (a)-(c), then it is well-known that h is isotopic to id# 
with respect to a neighbourhood of dBr\lZ <0 . See [HUH] for the 2-disk, 
and [6] for the cases (b) and (c). 

Let Q be the union of 7£<o with all the components of types (a)-(c). 
Then we can assume that h is fixed on Q. 

It also follows that Q is incompressible and every connected compo- 
nent Q' of Q contains some Nj. This implies that xiQ') < x(-^j) < 0- 
Then by (D) of Proposition 14.51 h is homotopic to id^ via a homo- 
topy fixed on Q. In particular, the restriction of h to every connected 
component B of type (d) is homotopic in B to ids relatively dB. □ 

Now we can complete Theorem II .71 

(1) It follows from the definition of TZ<o that dX consists of some 
regular leaves of Af, whence / is locally constant of OX. Moreover by 
Corollary 13.81 every connected component B of M \ 7?.<o and therefore 
of M \ X is either a 2-disk, or a cylinder, or a Mobius band. 

It is also easy to see that B contains critical points of /. Indeed, 
suppose B is either a 2-disk or a Mobius band. Since / is constant 
on dB, it follows that /|g is null-homotopic. Hence / must have local 
extremes in InlB. 

On the other hand, if B is a cylinder containing no critical points of 
/, then by the construction of X we should have that B C X which is 
impossible. 

Statement (2) is a particular case of (ii) of Claim [7721 

(3) We have to show that the inclusion i : S'(f,X) C S'(f) yields a 
bijection i : ir S'{f,X) w n S'(f). 

Claim 7.3. The map io : 7To«S'(/, X) — > 7i S'(f) is an epimorphism. 

Proof. Let h G S'(f). We have to show that h is isotopic in S'(f) to a 
diffeomorphism fixed on X. 

By (i) of Claim [7721 h preserves the foliation of Af on 7^<o. Hence by 
Proposition 16.11 applied to each critical component Ki, (i = 1, . . . , k), h 
is isotopic in S'(f) to a diffeomorphism fixed on some neighbourhood 
of 7£<o, so we can assume that h itself is fixed near 7£<o. 

Let Bi, (i = 1, . . . , q), be a connected component of X \ 7^ <0 . By 
the construction Bi is a cylinder being a union of regular leaves of Af 
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and containing no critical points of /. Choose an orientation for Bi. 
Then we can define a Hamiltonian flow H : Bi x R — > Bi of / on Bi 
whose orbits are leaves Aj belonging to Bi. Notice that h is fixed on 
some neighbourhood of OBi fl 7£<o and by (ii) the restriction of h to B 
is homotopic to id^ relatively dBi. Then by [TUJ Lm. 4.12] there exists 
a C°° function a : B$ — > M such that a = on some neighbourhood of 
9Sj fl 7?.<o and = H(x, for all x G 

Notice that <91?j fl 7^<o separates M. Then the map 

r„ +s n rr t * r / \ I -f/Yx, ta(x)), x e Bi, 

(7.1) a:MxI^M, afot ) = ^ >' 1 J' 

is an isotopy of h in to a diffeomorphism fixed on Bi. Applying 
this to each Bi we will made h fixed on all of X. □ 

Claim 7.4. i$ : 7ioS'(f,X) — > 7ro«S'(/) is a monomorphism. 

Proof. Let S' id (f) and S' id (f,X) be the identity path components of 
iS'(/) and S'(f,X) respectively. It is clear that S' id (f) = Si d (f). Hence 
an injectivity of %q means that 

S! d (f,X) = S'(f,X)nS! d (f) = S'(f,x)ns id (f). 

Evidently S' id (f,X) C S'(f, X) n S id (f)- 

Conversely, let h e A A )fliS i d(/), so h is fixed on some neighbour- 
hood of X and there exists an isotopy g t : M — > M in S(f) between 
h = id.M and hi = h. Then by Proposition 16.31 this isotopy can be 
made fixed on some neighbourhood of X. Hence h G S' id (f,X). □ 
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